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Abstract 

This  paper  assesses  the  incentive  and  award  fee  contracting  training  and  experiences  gained 
by  the  Department  of  Defense  (DoD)  Acquisition  Workforce  (DAW)  through  the 
implementation  of  various  incentive  arrangements  to  influence  more  favorable  performance 
outcomes.  The  researchers  developed  a  model  to  measure  current  and  expected  gaps 
between  training,  experience  and  knowledge.  Additionally,  the  model  correlates  training  and 
experience  with  knowledge  and  correlates  training  and  experience  with  performance 
outcomes.  The  researchers  used  a  survey  instrument  that  included  30  questions  to  capture 
observations  and  assessments  of  the  DAW  who  attended  advanced  classes  between  the 
period  of  2013  and  2015.  The  survey  provided  key  data  to  determine  the  presence  of  any 
noticeable  gaps  between  required  and  actual  levels  of  training,  experience  and  knowledge. 

Introduction 

Part  16  of  the  Federal  Acquisition  Regulation  (FAR)  describes  a  wide  variety  of 
contract  types  that  may  be  used  for  the  acquisition  of  goods  and  services  using  appropriated 
funds  in  the  Federal  Government.  The  contract  types  primarily  vary  according  to  (1) 
responsibilities  assumed  by  the  contractor  for  costs  of  performance  and  (2)  profit  incentives 
for  achieving  or  exceeding  specified  standards  or  goals.  The  contract  types  are  grouped  into 
two  broad  categories:  “fixed-price”  and  “cost-reimbursement.”  The  contract  types  range  from 
“firm-fixed-price,”  where  the  contractor  has  full  responsibility  for  performance  costs  and 
profits,  to  “cost-plus-fixed-fee,”  where  the  contractor  is  allocated  minimal  responsibility  for 
performance  costs  with  the  fee  (or  profit)  fixed  by  the  terms  of  the  contract.  In  between  these 
two  extremes  are  various  “incentive”  contract  types,  where  the  contractor’s  responsibility  for 
the  performance  costs  and  profit  are  adjusted  depending  on  the  actual  results  of  specific 
uncertainties  identified  at  the  time  of  contract  award.  Tables  1  and  2  present  a  summary  of 
the  FAR  descriptions  for  three  primary  fixed-price  type  contracts  and  three  cost- 
reimbursement  type  contracts. 
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Table  1,  Summary  of  Fixed  Price  Contract  Types 
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Background 

In  September  2010,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (USD[AT&L])  in  his  Better  Buying  Power  (BBP)  initiative  memos  clearly  expressed, 
as  a  matter  of  policy,  the  importance  of  properly  choosing  contract  types  as  a  way  of 
“aligning  the  incentives  of  the  government  and  contractor”  (Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics  [OUSD(AT&L)],  2010).  While  the 
September  2010  memo  BBP  1.0  emphasized  the  increased  use  of  Fixed-Price  Incentive 
Firm  Target  type  contracts,  BBP  2.0  refined  the  guidance  by  emphasizing  the  use  of  the 
“appropriate  contract  vehicle  for  the  product  or  services  being  acquired”  (e.g.,  “one  size 
does  not  fit  all”)  and  also  “focus[ed]  on  improving  the  training  of  management  and 
contracting  personnel  in  the  appropriate  use  of  all  contract  types”  (OUSD[AT&L],  2012). 

More  recently,  in  BBP  3.0,  there  was  even  more  refined  guidance:  “Employ  appropriate 
contract  types,  but  increase  the  use  of  incentive  type  contracts”  (OUSD[AT&L],  2015a). 
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Table  2.  FAR  Summary  of  Cost  Plus  Contract  Types 


Costal  US' -Incentive* 
Fee  (CPlf) 

Cost-Plus -Award  Fee 
(CPAF) 

Coit-Plui-FixMl-Fw 
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actual  costs)  would  be 
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estimated  cost  in  the  Contract  Part  l  the  Schedule.  Section  B  Supplies  or 
services  and  prices  costs. 

Contractor  Incentive 
(other  than 
maximizing  goodwill) 

Realizes  a  higher  fee 
by  completing  the  work 
at  a  Lower  cost  and  or 
by  meeting  other 
objective  performance 
targets. 

Realizes  a  higher  fee 
by  meeting  judgmental 
performance  standards. 

Realizes  a  higher  rate 
ofremm(Le..fet 
divided  by  total  cost)  as 
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In  a  report  entitled  Performance  of  the  Defense  Acquisition  System,  2015  Annual 
Report,  dated  September  16,  2015,  the  OUSD(AT&L)  concluded  “that  incentive  contracts 
(cost-plus-incentive-fee  and  fixed-price-incentive)  control  cost,  price,  and  schedule  as  well 
as,  or  better  than,  other  types — and  with  generally  lower  yet  fair  margins”  (OUSD[AT&L], 
2015b). 

Prior  to  the  September  2010  BBP  memo  cited  above,  the  Government  Accountability 
Office  (GAO;  2005)  reported  that  the  Department  of  Defense  (DoD)  had  paid  billions  in 
award  fees  regardless  of  acquisition  outcomes.  The  GAO  (2005)  audits  stated  the  award  fee 
criteria  were  subjective  and  faulty,  resulting  in  fees  awarded  for  marginal  performance. 

Even  though  a  number  of  corrective  actions  were  implemented,  they  still  failed  to 
address  a  key  area  of  training  and  experience  in  the  application  of  incentive  arrangements. 
The  report  by  the  GAO  stated  the  guidance  on  Award  Fees  has  led  to  better  practices  but  is 
not  consistently  applied  for  the  DoD.  The  audit  report  also  addressed  the  following  topics: 
programs  that  paid  fees  without  holding  contractors  accountable  for  achieving  desired 
acquisition  outcomes,  such  as  meeting  cost  and  schedule  goals  and  delivering  desired 
capabilities,  and  programs  that  paid  contractors  a  significant  portion  of  the  available  fee  for 
what  award  fee  plans  describe  as  “acceptable,  average,  expected,  good,  or  satisfactory" 
performance  when  the  purpose  of  these  fees  is  to  motivate  excellent  performance. 

The  shortfalls  of  incentive  contract  arrangements  are  well  documented  in  two  GAO 
studies  (GAO,  2005,  2009)  and  other  literature.  They  address  the  areas  of  cost  control, 
schedule,  management,  and  technical  performance.  Generally,  contractors  who  fail  to  meet 
incentive-related  goals  also  frequently  fail  to  meet  other  terms  and  conditions  of  the 
contract,  which  if  structured  properly,  would  carry  performance  penalties.  Some  of  the 
shortcomings  seen  in  the  application  of  incentive  contracts  led  to  several  clarifications  from 
senior  leadership  in  the  DoD,  including  additional  amplification  of  what’s  important  in  the 
BBP.  All  too  often,  the  contractors  continue  to  earn  high  fee  levels  on  late  or  cost  over  runs 
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because  the  criteria  tends  to  be  overly  subjective  and  the  outcomes  are  not  always  well 
written  with  clear  and  equivocal  objective  outcomes.  This  could  be  a  result  of  a  knowledge 
shortfall,  ineffective  training,  and/or  inexperience.  However,  no  research  has  been 
conducted  to  look  more  closely  at  the  root  cause  of  the  problem  in  incentive  contracting. 
This  study  hypothesized  that  knowledge,  experience,  and  training  could  be  contributing  to 
the  root  cause,  impeding  the  appropriate  use  of  contract  types  to  include  incentive  type 
arrangements. 

This  research  began  with  the  development  of  a  Research  Model  that  accounts  for 
the  variables  that  contribute  to  incentive  arrangement  failures.  The  research  pursuit  is 
intended  to  help  the  DoD  better  understand  the  level  of  experience,  knowledge  and  training 
required  for  the  effective  application  of  incentive  contract  arrangements.  The  researchers 
leveraged  the  Kirkpatrick  Learning  Model  (Figure  1),  specifically  Levels  11-Learning,  Ill- 
Behavior,  and  IV-Results. 


Kirkpatrick  Evaluation  Model 


Training  Environment 


-earning 

«  lAflfftff 

Cgs'iatc'i 


Work  Environment 


Figure  1.  Kirkpatrick  Evaluation  Model 

Kirkpatrick’s  four  levels  of  evaluation  consist  of 

•  Step  1 :  Reaction — How  well  did  the  learners  like  the  learning  process? 

•  Step  2:  Learning — What  did  they  learn?  (the  extent  to  which  the  learners 
gain  knowledge  and  skills) 

•  Step  3:  Behavior — What  changes  in  job  performance  resulted  from  the 
learning  process?  (capability  to  perform  the  newly  learned  skills  while  on  the 
job) 

•  Step  4:  Results — What  are  the  tangible  results  of  the  learning  process  in 
terms  of  reduced  cost,  improved  quality,  increased  production,  efficiency, 
etc.? 
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Research  suggests  that  as  much  as  90%  of  training  resources  are  spent  on  the 
design,  development,  and  delivery  of  training  events,  yet  only  yield  15%  on-the-job 
application  (Brinkerhoff,  2006).  The  learning  reinforcement  that  occurs  after  the  training 
event  actually  produces  the  highest  level  of  learning  effectiveness,  followed  by  activities  that 
occur  before  the  learning  event. 

Research  Model 
Training  Effectiveness  Model 

The  Training  Effectiveness  Model  (Figure  2),  would  assess  if  Knowledge,  Training, 
and  Experience  gaps  exist  among  Contracting  Officers,  Contract  Specialists,  Acquisition 
Professionals,  Program  Managers,  and  Deputy  Program  Managers.  The  primary  focus  of 
this  research  is  to  better  understand  the  experiences  (HI  A  and  H2A)  gained  through  the 
management  of  various  incentive  arrangements  and  determine  if  gaps  exist  in  experience 
levels,  and  whether  these  gaps  have  a  potential  causal  relationship  with  programmatic 
performance.  Training  received  through  the  DAU  and  other  training  (H1B  and  H2B)  were 
used  to  determine  the  presence  of  any  substantiated  gaps  and  their  influence  on 
performance  outcomes.  A  gap  analysis  would  confirm  the  disparity  between  current  and 
required  levels  of  knowledge,  training,  and  experience  to  achieve  desired  performance 
outcomes  performance.  For  Hypothesis  2,  a  correlation  was  used  to  determine  the  strength 
of  the  relationship  between  experience  and  training  and  its  potential  causal  relationship  to 
knowledge.  A  gap  analysis  and  correlation  was  also  conducted  to  determine  the  strength  of 
knowledge  supported  by  training  and  experience. 


tx&tf  terae  Gap 
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Figure  2.  Knowledge,  Experience,  and  Training  Effectiveness  Model 

A  Knowledge  Gap  is  defined  as  the  difference  between  what  Acquisition 
professionals  from  the  Program  Management  (PM)  and  Contracts  Management  (CM) 
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communities  observed  as  the  current  knowledge  levels  and  the  ideal  or  desired  knowledge 
level.  A  Knowledge  gap  was  measured  by  the  absolute  difference  of  the  arithmetic  mean  of 
the  observed  or  current  knowledge  level  and  the  required  knowledge  level  in  the  context  of 
incentive  and  award  fee  contracts. 

A  Training  Gap  is  defined  as  the  difference  between  what  the  Acquisition 
professionals  from  the  PM  and  CM  communities  observed  as  current  training  level  and  the 
ideal  or  desired  training  level.  A  Training  gap  is  measured  by  the  absolute  difference  of  the 
arithmetic  mean  of  the  observed  or  current  training  level  and  the  desired  or  required  training 
level  in  the  context  of  incentive  and  award  fee  contracts. 

An  Experience  Gap  is  defined  as  the  difference  between  what  the  Acquisition 
professionals  from  the  PM  and  CM  communities  observed  as  their  current  experience  levels 
and  the  ideal  or  desired  experience  level.  An  Experience  gap  is  measured  by  the  absolute 
difference  of  the  arithmetic  mean  of  the  observed  or  current  experience  level  and  the 
desired  or  required  experience  level  in  the  context  of  incentive  and  award  fee  contracts. 

Performance  is  the  overall  effectiveness  of  the  acquisition  professional  as  perceived 
by  the  Program  Managers  and  Contracting  Officer  groups.  Performance  is  defined  by  the 
eight  generic  attributes  (DoD,  2014)  that  program  managers  and  contracting  officers 
encounter  with  incentive  and  award  fee  contracts.  The  eight  attributes  are  cost  growth, 
program  schedule,  technical  requirements,  user  requirements,  technical  issues,  program 
risk,  cost  control,  and  contractor  performance.  The  validation  of  the  hypotheses  of  this 
research  depended  in  part  on  the  correlation  analysis  (at  a  minimally  acceptable  level  of 
significance  of  at  least  .05  or  p  <  0.05)  between  the  gaps  or  intervening  variables  and  the 
incentive  contracting  performance.  It  is  important  to  stress  that  a  correlation  coefficient  of 
any  magnitude  or  sign,  regardless  of  statistical  significance,  does  not  imply  causation 
(Emory  &  Cooper,  1991).  However,  the  study  explored  the  independent  variables  that  might 
in  some  way  influence  performance  outcomes.  Consistent  with  the  Kirkpatrick  Evaluation 
Model,  the  DoD  Acquisition  Workforce  survey  volunteers  provided  useful  information  about 
themselves  and  their  peers.  The  data  centered  on 

1 .  the  Effectiveness  of  classroom  training  received  by  the  respondent  and  its 
impact  on  job  performance  of  incentive  contracting  (Level  2) 

2.  job  performance  levels  achieved,  expected  and  targeted  with  respect  to 
incentive  and  award  fee  contracting  (Level  3) 

3.  performance  effects  of  perceived  gaps  in  knowledge,  training,  and  experience 
with  respect  to  incentive  and  award  fee  contracting  (Level  4) 

Research  Methodology 

The  intended  population  for  this  research  included  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA)  students  who  completed  the  Program  Management  (PM)  Office 
Course  (PMT  352)  and  Contracting  (CON)  for  Decision  Making  (CON  360)  from  each  of  the 
military  services,  DoD  agencies  and  included  support  contractors  between  the  period  of 
October  2013  and  December  2015.  Two  control  groups  were  surveyed:  a  Program 
Management  group  consisting  of  Program  Managers,  Deputy  Program  Managers,  and 
function  acquisition  leads;  and  the  Contract  Management  group  consisting  of  Procuring 
Contracting  Officers,  Administrative  Contracting  Officers,  Contract  Specialists,  and  contract 
support  administrators/staff.  A  statistical  analysis  was  performed  to  answer  the  following 
research  questions: 
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Research  Question  1 

What  are  the  relationships  among  training  gap,  experience  gap,  and  performance 
attributes  that  contribute  to  incentive  contract  arrangements  among  Defense  Acquisition 
Workforce  members  in  the  contracting  and  program  field? 

•  Hypothesis  1:  There  are  gaps  in  training  and  experience  that  affect 
performance  attributes  that  contribute  to  incentive  contract  arrangements 
among  Defense  Acquisition  Workforce  members  in  the  contracting  and 
program  field. 

o  Hypothesis  1A:  There  is  a  reliable  relationship  among  experience 
gaps  and  performance  attributes  among  contracts  and  program 
managers. 

o  Hypothesis  IB:  There  is  a  reliable  relationship  among  training  gaps 
and  performance  attributes  among  contracts  and  program  managers. 

Research  Question  2 

What  are  the  relationships  between  training  gaps,  experience  gaps,  and  knowledge 
gaps  that  contribute  to  incentive  contract  arrangements  among  Defense  Acquisition 
Workforce  members  in  the  contracting  and  program  field? 

•  Hypothesis  2:  There  are  differences  in  the  relationships  between  training 
gaps,  experience  gaps,  and  knowledge  gaps  that  contribute  to  incentive 
contract  arrangements  among  Defense  Acquisition  Workforce  members  in 
the  contracting  and  program  field. 

o  Hypothesis  2A :  There  is  a  reliable  relationship  among  experience 
gaps  and  knowledge  gaps  among  contracts  and  program  managers. 

o  Hypothesis  2B:  There  is  a  reliable  relationship  among  training  gaps 
and  knowledge  gaps  among  contracts  and  program  managers. 

The  researchers  used  a  web-based  survey  consisting  of  30  questions;  1,194 
individuals  from  the  program  management  track  responded,  and  946  individuals  from  the 
contracting  track  responded.  A  Beta  Testing  of  the  survey  instrument  was  conducted  with 
the  DAU  West  Contracting  and  PM  Department  faculty  prior  to  the  release  of  the  survey. 
The  researcher  performed  a  Cronbach’s  alpha  coefficients  assessment  that  resulted  in  a 
very  high  reliability  value  of  0.936,  p  <  .05. 

Data  Analysis 

This  research  measured  the  gaps  and  correlation  relationship  among  knowledge, 
training,  experience,  and  performance  gap  that  exist  in  incentive  and  award  fee  contracts. 
This  was  accomplished  by  assessing  the  students’  current  level  of  training,  knowledge  and 
the  experience  required  to  support  incentive  contract  arrangements,  compared  to  the  level 
of  knowledge,  training,  and  experience  required  to  achieve  expected  outcomes.  Data  from 
two  population  samples  (i.e.,  PMs  and  CMs)  identified  the  gaps. 

This  research  pursuit  was  based  on  empirical  data  in  order  to  assess  knowledge, 
training,  and  experience  gaps  that  could  be  influencing  the  success  or  failure  of  incentive 
and  award  fee  contract  arrangements.  This  accompanying  analysis  required  a  systematic 
method  that  accurately  described  the  relationships  among  the  independent,  dependent  and 
intervening  variables  measured. 
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Descriptive  correlation  research  was  the  most  suitable  choice  to  determine  accuracy 
(Isaac  &  Michael,  1977).  The  variables  of  this  study  consisted  of  two  independent  variables, 
one  intervening  variable,  and  one  dependent  variable,  as  follows: 

Independent  Variables 

•  Observed  vs.  Required/Expected  Incentive  Contracts  Experience  Gaps 

•  Observed  vs.  Required/Expected  Incentive  Contracts  Training  Gaps 
Intervening  Variables 

•  Observed  vs.  Required/Expected  Incentive  Contract  Knowledge  Gaps 
Dependent  Variables 

•  Performance  Attributes 

Hypothesis  1:  There  are  gaps  in  training  and  experience  that  affect  performance  attributes 
that  contribute  to  incentive  contract  arrangements  among  Defense  Acquisition  Workforce 
members  in  the  contracting  and  program  field. 

Table  3  provides  the  descriptive  statistics  of  the  knowledge,  training,  and  experience 
gaps.  It  also  shows  the  mean,  median,  mode,  and  standard  deviation  of  each  variable.  The 
gaps  for  both  the  PM  and  CM  were  moderately  strong  at  3.8  to  4.3  on  a  6-point  Likert  scale. 
The  PMs  and  CMs  indicated  noticeable  gaps  in  training,  experience,  and  knowledge. 

Table  3.  Descriptive  Statistics  for  CM/PM  Knowledge,  Training,  and  Experience 

Gaps 


CM  Know 

Gap 

CM  Train 

Gap 

cm&sc 

Gap 

PM  Know 

Gap 

PM  Train 

Gap 

PM£ap 

Gap 

N 

424 

424 

424 

424 

424 

424 

Mean 

3.77S3 

3. Si  13 

4  1061 

3  945* 

39033 

4  3302 

Median 

4  0000 

4  0000 

50000! 

4  0000 

40000 

5  0000 

Mode 

600 

600 

6  00 

600 

600 

600 

Std  Deviation 

1  77396 

1.92503 

1  SS952 

1  67710 

I  72249 

1  63S61 

Variance 

3,147 

3.331 

3.570 

2.813 

2967 

2685 

Minimum 

1  OO 

1  00 

1  00 

1.00 

1.00 

1  00 

Maximum 

600 

600 

600 

600 

600 

600 

Table  4  shows  the  Pearson’s  r  analysis  of  the  independent  variable,  training  gap, 
and  the  dependent  variable  of  performance  by  PMs.  All  nine  performance  attributes  had  a 
moderately  low  correlation  that  ranged  from  .359  to  .441,  p  <  .000.  All  of  these  correlations 
had  a  significant  confidence  level  at  the  0.01  level. 
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Table  4.  Results  of  the  Pearson’s  r  Correlation  Between  Program  Managers 

Training  Gap  and  Performance  Attributes 


Variable* 

N 

Pttnon  i  r 

Sig. 

Cost  Growth  -  Training  Gap 

506 

.441 

.000 

Program  Schedule  -  Training  Gap 

506 

.407 

.000 

Tech  Requirements  -  Training  Gap 

506 

.199 

.000 

User  Requirements-  Training  Gap 

506 

.181 

,000 

Technical  Issues  -  Training  Gap 

506 

.371 

,000 

Program  Ride  -  Training  Gap 

506 

,159 

.000 

Cost  Control  -  Training  Gap 

506 

.401 

-000 

Contractor  Performance  -  Training  Gap 

506 

.1$) 

.000 

Sate.  *  Correlation  is  significant  at  the  0.01  level  (two-tailed) 


Table  5  shows  the  Pearson’s  r  analysis  of  the  independent  variable,  training  gap, 
and  the  dependent  variable  of  performance  by  CMs.  Cost  growth  indicated  a  very  strong 
correlation  of  r  =  .934,  p  <  .000.  The  remaining  seven  performance  attributes  had  a 
moderately  high  correlation  ranging  from  .592  to  .717,  p  <  .000.  All  of  these  correlations  had 
a  significant  confidence  level  at  the  0.01  level. 

Table  5.  Results  of  the  Pearson’s  r  Correlation  Between  Contracts  Managers 

Training  Gap  and  Performance  Attributes 


Variable* 

N 

Pearson's  r 

Sig. 

Cost  Growth  -  Training  Gap 

r: 

.914 

.000 

Program  Schedule  -  Training  Gap 

m 

.690 

.000 

Tech  Requirements  -  Training  Gap 

m 

.694 

.000 

User  Requirements-  Training  Gap 

r: 

.622 

.000 

Technical  Issues  -  Training  Gap 

172 

.669 

.000 

Program  Risk  -  Training  Gap 

172 

.592 

.000 

Cost  Control  -  Training  Gap 

r: 

.717 

.000 

Contractor  Performance  -  Training  Gap 

r: 

.659 

.000 

Sole.  ’Correlation  is  significant  at  the  0.01  level  (two-tailed) 


Table  6  shows  the  Pearson’s  r  analysis  of  the  independent  variable,  experience  gap, 
and  the  dependent  variable  of  performance  by  PMs.  Cost  growth  indicated  a  moderately 
strong  correlation  r  =  .728,  p  <  .000.  The  remaining  seven  performance  attributes  had  a 
moderate  correlation  ranging  from  .466  to  .584,  p  <  .000.  All  of  these  correlations  had  a 
significance  at  the  0.01  level. 
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Table  6.  Results  of  the  Pearson’s  r  Correlation  Between  Program  Managers 
Experience  Gap  and  Performance  Attributes 


Variable* 

N 

Pearson's  r 

Sig. 

C o*l  Growth  —  Experience  Gap 

506 

.000 

Program  Schedule  -  Experience  Gap 

506 

.53  2 

.000 

Tech  Requirement*  -  Experience  Gap 

506 

.511 

.000 

User  Requirements-  Experience  Gap 

506 

.466 

.000 

Technical  Issues  -Experience  Gap 

506 

4&3 

ooo 

Program  Risk  —  Experience  Gap 

506 

.549 

*000 

Cost  Control  -  Experience  Gap 

506 

.5S4 

.000 

Contractor  Performance  -Experience  Gap 

506 

.546 

ooo 

Note.  *  Correlation  ii  significant  At  the  0.01  level  (two-tailed) 


Table  7  shows  the  Pearson’s  r  analysis  of  the  independent  variable,  experience  gap, 
and  the  dependent  variable  of  performance  by  CMs.  Program  Risk  indicated  a  strong 
correlation  of  r  =  .911,  p  <  .000.  The  remaining  seven  performance  attributes  had  a 
moderately  strong  correlation  ranging  from  .782  to  .894,  p  <  .000.  All  of  these  correlations 
had  a  significance  at  the  0.01  level. 

Table  7.  Results  of  the  Pearson’s  r  Correlation  Between  Contracts  Managers 
Experience  Gap  and  Performance  Attributes 


Variables 

N 

Pearson's 

r 

Sig. 

Ceil  Growth  -  Experience  Gap 

372 

.874 

.000 

Program  Schedule  -  Experience  Gap 

372 

.873 

.000 

Tech  Requirements  -  Experience  Gap 

372 

.851 

.000 

User  Requirement*-  Experience  Gap 

372 

.823 

.000 

Technical  Issues  -Experience  Gap 

372 

.854 

.000 

Program  Risk  -  Experience  Gap 

372 

.91 1 

.000 

Cost  Control  -  Experience  Gap 

372 

.894 

.000 

Contractor  Performance  -  Experience  Gap 

372 

.782 

.000 

Note.  *  Correlation  is  significant  at  the  0,01  level  <two*taitcd) 


Table  8  shows  a  mean  gap  analysis  of  current  and  should  be  performance  levels  of 
incentive  and  award  fee  contracts  by  the  CM  respondents.  The  scale  measures  the  various 
levels  of  understanding  of  outcomes  that  can  be  achieved  through  incentive  arrangement. 
On  a  Likert  scale  of  1-6,  1  represents  knowing  the  characteristics  of  an  incentive 
arrangement,  2  represents  understanding  the  benefits,  3  represents  applying  the 
mechanics,  4  represents  analyzing  the  risks,  5  represents  evaluating  outcomes,  and  6 
represents  creating  and  developing  suitable  contracts. 
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Table  8.  Results  of  a  Mean  Gap  Analysis  Between  Contract  Managers  Current 

and  Should  Be  Performance  Level 


Yambkt 

N 

Current 

Mean 

Current 

SD 

Sboddbe 

Mean 

should 

beSD 

Mean 

Gap 

Program  Manager 

m 

3.0968 

1.7832 

4.1184 

1.7721 

1.0216 

Deputy  Program 
Manager 

m 

3.0370 

1.7899 

4.0SW 

1.711S 

1.0494 

Contra  ting  Officer 

127 

4.0185 

2,0140 

5.1804 

1 5112 

1J619 

Contract  Specials 

169 

-  >!:< 

1.9019 

4.7015 

1.7621 

U590 

The  CM  respondents  identified  a  gap  greater  than  1 .0  for  all  four  groups.  The 
greatest  gaps  were  contract  specialists,  with  a  1 .3590,  and  contracting  officers,  with  a 
1.1619.  The  contracting  officers  could  believe  that  among  the  contracting  officers  and 
contracting  specialists,  performance  levels  should  result  in  more  meaningful  performance 
outcomes. 

Table  9  shows  a  mean  gap  analysis  of  current  and  should  be  performance  levels  of 
incentive  and  award  fee  contracts  by  the  PM  respondents.  The  PM  respondents  identified  a 
gap  greater  than  1 .0  for  each  of  the  groups.  The  greatest  gaps  were  deputy  program 
managers,  with  a  2.0384,  and  contract  specialists,  with  a  1 .4396.  Program  managers  could 
believe  that  among  deputy  program  managers  and  contract  specialists,  performance  levels 
should  result  in  more  meaningful  performance  outcomes. 

Table  9.  Results  of  a  Mean  Gap  Analysis  Between  Program  Managers  Current 

and  Should-Be  Performance  Level 


Variables 

N 

Current 

Mean 

Curreni 

SD 

Should 

be  Mean 

Should 

be  SD 

Mean 

Gap 

Program 

Manager 

386 

3.0200 

1.7777 

4.1036 

1.7756 

1.0836 

Deputy 

Program 

Manager 

383 

2.0112 

1.5380 

4.0496 

1.7351 

2.0384 

Contracting 

Officer 

343  | 

3,9510 

2.0372 

5.1590 

1.5382 

1.2080 

Contract 

Specialist 

327 

3.2416 

1.9019 

4.6812 

1.7604 

1.4396 

The  results  of  the  multiple  regression  analysis  are  shown  in  Table  10.  The  analysis 
was  conducted  with  the  independent  variable  of  CM  training  means  and  dependent 
variables  of  performance  attributes.  The  constant  determines  the  overall  effect  of  the 
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variables  on  the  performance  attributes.  The  multiplier  R-value  was  0.935,  p  <  0.000;  and 
the  multiple  R-squared  was  0.872,  suggesting  that  the  independent  variable,  CM  training, 
explains  the  93.52%  of  the  variances  in  performance  attributes.  Multiple  regression  analysis 
was  used  to  evaluate  the  strength  of  the  relationship  among  the  independent  and  dependent 
variables.  The  regression  model  summary  and  coefficients  implies  that  CM  training  had  a 
strong  positive  correlation  with  performance  attributes. 

Table  10.  Multiple  Regression  of  CM  Training  as  a  Function  of  the  Performance 

Attributes 


Dependent  Variables 

B 

SEB 

Beta 

1 

p< 

(Constant) 

- SET" 

mr 

0.000 

Cost  Growth 

1,490 

0.05S 

0.937 

25.775 

0.000 

Program  Schedule 

-0.015 

0.054 

-0.010 

-0.276 

0.783 

Tech  Requirements 

-0.015 

0.067 

•0.010 

■0.230 

0.S1S 

User  Requirements 

0.051 

0.051 

0.036 

1.011 

0.312 

Technical  Issues 

-0.015 

0.054 

-0.010 

-0.231 

0.780 

Program  Risk 

-0,033 

0.053 

-0.053 

-1.591 

0.113 

Cost  Control 

0.110 

0.053 

0.076 

2.050 

0,041 

Contractor  Performance 

-0.049 

0.046 

-0.033 

-0.043 

0.298 

Independent  Variable: 

Multiple  R  =  0.935 

R-squared  =  0.872 

p<  0.000 

CM  Training  Mean 

The  results  of  the  multiple  regression  analysis  are  shown  in  Table  11.  The  analysis 
was  conducted  using  the  independent  variable  of  CM  experience  means  and  dependent 
variables  of  performance  attributes.  The  constant  determines  the  overall  effect  of  the 
variables  on  the  performance  attributes.  The  multiplier  R-value  was  0.920,  p  <  0.000;  and 
the  multiple  R-squared  was  0.844,  suggesting  that  the  independent  variable,  PM 
experience,  explains  the  92.0%  of  the  variances  in  performance  attributes.  Multiple 
regression  analysis  was  used  to  evaluate  the  strength  of  the  relationship  among  the 
independent  and  dependent  variables.  The  regression  model  summary  and  coefficients  can 
be  interpreted  that  PM  experience  had  a  strong  positive  correlation  with  performance 
attributes. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-430- 


Table  11.  Multiple  Regression  of  CM  Experience  as  a  Function  of  the  Performance 

Attributes 


Dependent  Variables 


B  SEB  Beta  t  p< 


(Constant) 

0.256 

0.096 

2.459 

0.014 

Cost  Growth 

0.258 

0.081 

0.233 

2.960 

0.003 

Program  Schedule 

0.074 

0.056 

0.075 

1.312 

0.190 

Tech  Requirements 

0.013 

0.071 

0.013 

0.189 

0.850 

User  Requirements 

-0.015 

0.050 

-0.015 

•0.293 

0.770 

Technical  Issues 

0.070 

0.049 

0.071 

1.441 

0.150 

Program  Risk 

0.369 

0.056 

0.361 

6.536 

0.000 

Cost  Control 

0.290 

0.050 

0.285 

5.815 

0.000 

Contractor  Performance 

-0.062 

0,053 

-0.064 

-1.186 

0.236 

Independent  Variable:  Multiple  J* - 0.900  R-squared  =0.544  p< 0.000 
CM  Experience  Mean 


The  results  of  the  multiple  regression  analysis  are  shown  in  Table  12.  The  analysis 
was  conducted  using  the  independent  variable  of  PM  training  means  and  dependent 
variables  of  performance  attributes.  The  constant  determines  the  overall  effect  of  the 
variables  on  the  performance  attributes.  The  multiplier  R-value  was  0.450,  p  <  0.000;  and 
the  multiple  R-squared  was  0.203,  suggesting  that  the  independent  variable,  PM  training, 
explains  the  45.0%  of  the  variances  in  performance  attributes.  Multiple  regression  analysis 
was  used  to  evaluate  the  strength  of  the  relationship  among  the  independent  and  dependent 
variables.  The  regression  model  summary  and  coefficients  imply  that  PM  training  has  a 
weak  positive  correlation  with  performance  attributes. 
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Table  12.  Multiple  Regression  of  PM  Training  as  a  Function  of  the  Performance 

Attributes 


Dependent  Variables  B  SEB  Beta  t  p< 


(Constant) 

OS  IS 

0.231 

3.533 

0.000 

Cost  Growth 

0.586 

0.16S 

0.446 

3.488 

0.001 

Program  Schedule 

0.159 

0.162 

0.125 

0.983 

0.326 

Tech  Requirements 

0.066 

0.196 

0.052 

0.336 

0.737 

User  Requirements 

0,067 

0.159 

0.052 

0.422 

0.6" 

Technical  Issues 

-0.226 

0.159 

-0.173 

•1.418 

0.157 

Program  Risk 

•0.184 

0.147 

*0.145 

*1252 

0.211 

Cost  Control 

0.072 

0.144 

0.056 

0.499 

0.618 

Contractor  Perfomtance 

0,029 

0.132 

0.023 

0.223 

0.824 

Independent  \anable:  Multiple  /f  -  0.450  R-squared  =  0.203  p<  0.000 

PM  Training  Meai( 


The  results  of  the  multiple  regression  analysis  are  shown  in  Table  13.  The  analysis 
was  conducted  considering  the  independent  variable  of  PM  experience  means  and 
dependent  variables  of  performance  attributes.  The  constant  determines  the  overall  effect  of 
the  variables  on  the  performance  attributes.  The  multiplier  R-value  was  0.730,  p  <  0.000; 
and  the  multiple  R-squared  was  0.534,  suggesting  that  the  independent  variable,  PM 
experience,  explains  the  73.0%  of  the  variances  in  performance  attributes.  Multiple 
regression  analysis  was  used  to  evaluate  the  strength  of  the  relationship  among  the 
independent  and  dependent  variables.  The  regression  model  summary  and  coefficients 
imply  that  PM  experience  has  a  moderately  positive  correlation  with  performance  attributes. 
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Table  13.  Multiple  Regression  of  PM  Experience  as  a  Function  of  the  Performance 

Attributes 


Dependent  Variables  S  SE3  Beta  f  p< 


(Constant) 

-0.543 

0.296 

•I.S3J 

0.068 

Cost  Growth 

0.975 

0.095 

0.701 

10.274 

0.000 

Program  Schedule 

-0.097 

0.103 

-0.071 

-0.942 

0.347 

TechRequumects 

0.014 

0.119 

0.011 

0.116 

0.908 

User  Requirements 

0.059 

0.0S7 

0.01$ 

0.675 

0.500 

Technical  Issues 

-0.006 

0.0S6 

-0.004 

•0.066 

0.948 

Program  Risk 

0.073 

0.107 

0.053 

0.689 

0.491 

Cost  Control 

0.060 

0.094 

0,014 

0.636 

0.525 

Contractor  Performance 

-0.049 

0.090 

-0.036 

•0.549 

0,583 

Independent  \  amble:  Mu]tipk  ^  =  07 jo  R*p«|=  0.534  p<  0.000 

PM  Experience  Mean 


Hypothesis  2:  There  are  differences  in  the  relationships  between  training  gaps,  experience 
gaps,  and  knowledge  gaps  that  contribute  to  incentive  contract  arrangements  among 
Defense  Acquisition  Workforce  members  in  the  contracting  and  program  field. 

•  Hypothesis  2A:  There  is  a  reliable  relationship  among  experience  gaps  and 
knowledge  gaps  among  contracts  and  program  managers. 

•  Hypothesis  2B:  There  is  a  reliable  relationship  among  training  gaps  and 
knowledge  gaps  among  contracts  and  program  managers. 

A  Pearson’s  Correlation  (i.e.,  linear  correlation)  analysis  was  conducted  between  the 
training  gaps,  experience  gaps,  and  the  knowledge  gaps  and  is  recorded  in  Table  14.  For 
CM  knowledge  gap  and  training  gap,  a  strong  correlation  of  r  =  0.898,  p  <  0.000  is 
supported.  CM  knowledge  gap  and  CM  experience  gap  have  a  strong  correlation  of  r  = 
0.893,  p  <  0.000.  For  program  managers  (PM)  knowledge  gap  and  training  gap,  a  strong 
correlation  of  r  =  0.843,  p  <  0.000  is  supported.  PM  knowledge  gap  and  CM  experience 
gap  have  a  strong  moderate  correlation  of  r  =  0.767,  p  <  0.000.  Pearson's  r  correlation 
results  for  CM  and  PM  training,  experience,  and  knowledge  gaps  are  presented  in  Table  14. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-433- 


Table  14.  Results  of  the  Pearson’s  r  Correlation  Between  Training  Gap, 

Experience  Gap,  and  Knowledge  Gap 


Variables 

N 

Pearson's  r 

Sig. 

CM  Knowledge  Gap  -  CM  Training  Gap 

445 

.898 

.000 1 

CM  Knowledge  Gap  -  CM  Experience  Gap 

445 

.893 

.000 1 

PM  Knowledge  Gap  -  PM  Training  Gap 

369 

.843 

.000 

PM  Knowledge  Gap  -  PM  Experience  Gap 

369 1 

.767 

.000 1 

Note.  *Correlation  is  significant  at  the  0.05  level  (two-tailed) 


The  multiple  R-squared  was  0.898  and  .893,  suggesting  that  the  two  independent 
variables,  experience  gap  and  training  gaps,  explain  89.8%  and  89.3%  of  the  variances  in 
knowledge  gap,  respectfully. 

Multiple  regression  analysis  was  used  to  evaluate  the  strength  of  the  relationship 
among  the  independent  and  dependent  variables.  The  regression  model  summary  and 
coefficients  can  be  interpreted  such  that  knowledge  gap  and  experience  gap  have  a  strong 
positive  correlation  with  knowledge  gap. 

Table  15  shows  the  multiple  regression  of  knowledge  gap  as  a  function  of  CM 
training  gap  and  CM  experience  gap.  There  is  a  strong  correlation  between  the  variable  of 
r  =  .925,  p  <  0.000.  This  implies  that  when  the  CM  training  gap  aligns  with  experience 
gaps,  knowledge  gap  will  be  optimized.  Hypothesis  H2is  supported  for  CM  Training  gaps, 
CM  Experience  gaps  and  Knowledge  gaps. 

Table  15.  Multiple  Regression  of  Knowledge  Gap  as  a  Function  of  Training  Gap 
and  Experience  Gap  for  the  Contracting  Group 


XnMc 

B 

SEB 

Beta 

t 

P< 

(GxBUflt) 

.160 

m 

L7S9 

.074 

CM  Training  Gap 

,430 

,035 

*45S 

12,251 

,000 

CM  Experience  Gap 

.490 

.037 

.497 

13.316 

,000 

Dependcm  Variable: 

Multiple  It  =  Q.9I5 

R-tqtured  =  0.555 

p  <0.000 

KoowtedgcCap 

Note,  "Correlation  iv  significant  at  the  0.05  level  (two-tailed) 


Table  16  shows  the  multiple  regression  of  knowledge  gap  as  a  function  of  PM 
training  gap  and  PM  experience  gap.  There  is  a  moderately  strong  correlation  between  the 
variable  of  r  =  .785,  p  <  0.000.  This  implies  that  when  the  PM  training  gap  aligns  with 
experience  gap,  knowledge  gap  will  be  moderately  optimized.  Hypothesis  H2  is  supported 
for  PM  Training  Gaps,  PM  Experience  gaps  and  Knowledge  gaps. 
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Table  16.  Multiple  Regression  of  Knowledge  Gap  as  a  Function  of  Training  Gap 
and  Experience  Gap  for  the  Program  Management  Group 


Yxublc 

B 

SEB 

Bela  f 

PK 

(Cocrtm) 

m 

Ml 

5663 

.000 

PMTramin^Gap 

.554 

.066 

.45*  tm 

.000 

Experience  Gap 

.354 

.064 

.497  JJS9 

.000 

Dependent  V ruble 

Knowledge  Gap 

Muhipk  R  - 

0.7*5 

fUquared  -  0.616 

p<  0,000 

S'ott.  ’Correlation  is  significant  at  the  0.05  level  (two-tailed) 


Research  Findings 
Summary 

With  respect  to  incentive  contracting  arrangements, 

•  Moderate  (self-assessed)  gaps  exist  in  knowledge,  training  and  experience. 

•  The  experience  gap,  vice  the  training  gap,  is  perceived  to  be  more  closely 
related  to  performance  outcomes,  and  Contract  Managers  see  training  to  be 
more  closely  related  than  do  Program  Managers. 

•  Training  and  experience  are  highly  correlated  to  knowledge. 

•  The  current  state  of  skill-sets  is  more  about  the  mechanics,  while  the  desired 
state  is  more  about  creating  and  developing  suitable  contracts. 

•  There  are  only  a  few  significant  differences  between  Contract  Managers  and 
Program  Managers  regarding  assessments  of  knowledge,  training,  and 
experience. 

Discussion 

Both  Program  Manager  and  Contract  Manager  Respondents  reported  moderate 
knowledge,  training,  and  experience  gaps  amongst  their  organization  personnel  who 
implement  incentive  contract  arrangements. 

Knowledge  associated  with  contractor  incentives,  and  associated  incentive 
contracting  approaches  and  techniques,  are  necessary  elements  in  addressing  performance 
obstacles.  Our  research  indicates  that  both  Program  Managers  and  Contract  Managers 
perceive  a  strong  relationship  between  both  the  observed  training  gap  and  experience  gap 
to  the  observed  knowledge  gap. 

Our  research  suggests  that  experience,  rather  than  training,  is  more  closely  identified 
with  performance  issues  using  incentive  arrangements.  More  specifically,  with  a  noted 
exception,  respondents  did  not  strongly  relate  the  observed  training  gap  as  impacting  skill- 
sets  needed  to  address  acquisition  obstacles  (e.g.,  “Unexpected  Cost  Growth,”  “Changes  in 
Program  Schedules,”  “Changes  in  Technical  Requirements,”  “Changes  in  User 
Requirements,”  Technical  Issues,”  and  “Program  Risk”).  The  exception  was  Contract 
Manager  Respondents  who  more  strongly  identified  the  relationship  between  training  gaps 
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and  meeting  performance  challenges  but  reached  the  strongest  only  for  the  cost-related 
acquisition  obstacles  (e.g.,  “Unexpected  Cost  Growth”  and  “Cost  Control”). 

Our  survey  asked  Contract  Managers  (Contracting  Officers  and  Contract  Specialists) 
and  Program  Managers  (Program  Managers  and  Deputy  Program  Managers)  to  self-assess 
their  “current”  and  “should  be”  levels  of  performance  in  working  with  incentive  contract 
arrangements,  thereby  allowing  visibility  of  any  perceived  gaps.  Our  research  also  asked 
Contract  Managers  and  Program  Managers  to  assess  each  other. 

Using  a  6-point  Likert  scale,  Contract  Managers  self-assessed  themselves  near  the 
middle  of  the  range  and  indicated  they  needed  to  increase  performance  levels,  significantly 
from  the  current  state.  Program  Manager  assessments  of  Contract  Manager  performance 
level  were  very  closely  aligned  with  those  of  the  Contract  Manager  self-assessments 
previously  mentioned  and  corroborated  what  the  research  confirmed. 

The  Program  Manager  self-assessments  were  very  similar  to  both  the  Contract 
Managers’  self-assessments  and  the  Contract  Managers’  assessments  of  Program 
Managers  except  when  it  came  to  the  assessment  of  Deputy  Program  Managers,  where 
there  was  more  than  a  2-point  gap  (Current  =  2.0  or  “understands  benefits,”  while  Should-be 
=  4.05  or  “analysis  of  risks”). 

As  expected,  a  strong  relationship  was  noted  between  the  observed  training  and 
experience  gaps  and  knowledge  gaps,  suggesting  that  reducing  training/experience  gaps 
could  also  reduce  any  knowledge  gaps. 


Qualitative  Data 

The  respondents  provided  over  6,000  comments,  and  the  following  is  a 
representative  sample  of  the  strengths  and/or  shortcomings  of  the  application  of  incentive 
contracts. 

Knowledge 

•  “Contractor  outcomes — profit/fee — were  higher  than  they  should  have  been 
because  personnel  routinely  failed  to  hold  contractors  to  the  criteria  found  in 
the  Award  or  Incentive  plans.” 

•  “I  think  people  don’t  want  to  use  contract  types  they  don’t  fully  understand.” 

•  “I  strongly  believe  that  incentive  contracts  are  necessary  at  my  command  and 
have  recommended  them  after  being  brought  on.  It  was  acknowledged  that 
this  is  the  best  contract  type  in  the  interest  of  our  program,  but  overly 
complicated  and  burdensome  given  the  lack  of  training  of  our  staff.” 

•  “KO  was  unable  to  write  the  outcomes  to  meet  my  program  outcomes.  I  was 
pushed  to  use  FFP  or  CPFF  or  IDIQ  arrangements  because  KO  did  not  fully 
understand  the  formation  of  award  fee  contracts.” 

•  “Not  sure  all  of  my  people  are  good  at  thinking  through  how  different  incentive 
structures  will  cause  the  contractor  to  behave.  Incentives  other  than  cost  are 
particularly  tricky  and  I  generally  shy  away  from  them  because  I  sense  that 
the  contractor  will  'outfox'  us  and  we  will  end  up  regretting  the  structure  down 
the  road.” 

Training 

•  “People  had  the  training,  but  did  not  understand  how  to  use  it  in  their  duties.” 

•  “Most  time  is  spent  trying  to  apply  mechanics  of  type  rather  than  truly 
implementing  meaningful  measures.” 
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•  “The  biggest  problem  is  developing  meaningful  criteria.  Services  acquisition 
are  very  hard  to  ‘incentivize.’  The  wording  of  the  meaningful  criteria  is  the 
bigger  problem  than  coming  up  with  10,  20,  30  percent  incentive.  That  is 
simple  math.” 

•  “Because  we  don't  have  good  command  level  training  for  how  to  administer 
these  overly  complex  contracts,  everyone  struggles  to  efficiently  do  their 
work.” 

•  “Lack  of  training  is  the  biggest  problem.” 

•  “We  do  not  receive  enough  training  to  understand  these  concepts  enough  to 
execute  effectively.” 

Experience 

•  “Experience  is  the  main  driver  of  shortfalls.  Not  everybody  is  in  situations 
where  they  are  using  CPAF,  FPIF,  or  CPIF  contracts  regularly.” 

•  “This  lack  of  experience,  understanding  and  training  makes  it  very  difficult  to 
effectively  utilize  these  contract  types.” 

•  “Nobody  seems  to  know  what  to  do,  if  and  when  we  use  it  and/or  the  required 
information  is  not  passed  to  the  field. 

•  “Experience  seems  to  drive  individuals  to  contracts  that  they  are  familiar 
with.” 

•  “Lack  of  experience  with  multiple  contract  types  can  cause  under¬ 
performance.” 

•  “The  lack  of  experience  outs  (place)  the  government  at  a  disadvantage  in 
execution  of  these  types  of  contracts.” 

•  “There’s  just  no  substitute  for  experience,  not  IQ,  not  education.  With 
experience  comes  intuition,  and  it’s  intuition  that  recognizes  what  flies  and 
what  doesn’t.  Oftentimes,  people  are  thrust  into  programs  and  projects  for 
which  they  lack  a  basis  for  making  informed  decisions  about  the  future.” 

Recommendations 

1 .  Ensure  incentive  contract  development  training  be  reinvigorated  for  PMs  and 
CMs  to  fully  understand  the  fundamental  principles  and  benefits  of  incentive 
contracts  as  well  as  provide  a  rigorous  setting  where  they  can  practice 
designing  incentive  contracts  and  applying  them. 

2.  Identify  the  lessons  learned  from  successful  and  unsuccessful  incentive 
arrangements  within  the  past  10  years  and  make  them  available  across  the 
defense  acquisition  workforce;  incorporate  these  to  the  greatest  extent 
possible  into  existing  courseware. 

3.  Produce  an  incentive  and  award  fee  guidebook  that  addresses  the 
performance  attributes  identified  in  this  study  and  address  the  question  “How 
do  I  write  an  incentive  arrangement?”  to  include  incorporating  performance 
attributes  that  lead  to  achievement  of  the  PM’s  goals  and  outcomes. 

4.  Ensure  the  appropriate  acquisition  workforce  qualification  competencies  (or 
any  variation  thereof)  incorporate  the  key  standards  that  address  incentive 
and  award  fee  contract  proficiencies  that  are  assignment  specific. 
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5.  Recommend  that  the  Functional  Integrated  Product  Team  (FIPT)  explore  a 
wide  range  of  competencies  that  are  specifically  tuned  to  the  implementation 
of  incentive  arrangements  for  PMs  and  CMs  in  assignment  specific  positions. 

6.  Conduct  a  follow  on  study  to  address  the  relationship  between  knowledge, 
performance,  and  the  applicable  regulations. 
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:  PRACTICE  wcmMiHwnoN 


3AU  Purpose  of  this  Study 

V 

•  Assess  the  incentive  and  award  fee  contracting  training,  experiences,  and 
knowledge  gained  by  the  DOD  Acquisition  Workforce  (DAW),  (i.e., 
Department  of  Defense  (DOD)  Program  Managers  (PMs),  Deputy  PMs 
(DPMs),  Contracting  Officers  (KOs),  and  Contract  Specialist  (CS)),  through 
the  implementation  of  various  contractual  incentive  arrangements  to 
influence  more  favorable  performance  outcomes. 

•  Identify  the  presence  of  any  noticeable  gaps  between  required  and  actual 
levels  of  training,  experience  and  knowledge  in  incentive  arrangements. 

•  Identify  opportunities  to  improve  the  training  and  experience  for  the  DAW 
through 

-  Additional  Training 

-  Improved  training  and/or 

-  Additional  experience  requirements  to  support  assignment  specific  DAW 
members. 
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KIRKPATRICK  EVALUATION  MODEL 


Training  Environment 


Level  1 : 


Learning  Event 


Reactions 

•  Learner 

•  Client 


I 


Kirkpatrick’s  four  levels  of  evaluation 
consist  of: 

Step  1 :  Reaction  -  How  well  did  the  learners 
like  the  learning  process? 

Step  2:  Learning  -  What  did  they  learn? 

(e.g.  the  extent  to  which  the  learners  gain 
knowledge  and  skills) 


Work  Environment 


Level  3: 
Results 

Performance 

Financial 


r 


Level  3: 
Job  Behavior 

•  Learner 

*  Organization 


Step  3:  Behavior  -  What  changes  in  job 
performance  resulted  from  the  learning 
process?  (e.g.  capability  to  perform  the 
newly  learned  skills  while  on  the  job) 

Step  4:  Results  -  What  are  the  tangible 
results  of  the  learning  process  in  terms  of 
reduced  cost,  improved  quality,  increased 
production,  efficiency,  etc.? 
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£AU  RESEARCH  MODEL 


I 

I 

I 

I 

I 


/ 

1 

1 

1 

1 

I 

H1B 

Performance 


Research  focus:  Better  understand  the 
experiences  (Ex)  (i.e.  HI  A  and  H2A)  gained 
through  the  management  of  various  incentive 
arrangements  and  determine  if  gaps  exist  in 
experience  levels,  and  whether  these  gaps  have  a 
potential  causal  relationship  with  programmatic 
performance. 

Training  (Tr)  (i.e.  H1B  and  H2B)  was  used  to 
determine  the  presence  of  any  substantiated  gaps 
and  their  influence  on  performance  outcomes. 

A  gap  analysis  would  confirm  the  disparity  between 
current  and  required  levels  of  knowledge  (Kn), 
training  (Tr),  and  experience  (Ex)  to  achieve  desired 
performance  outcomes  performance. 


Independent  Variable 


Dependent  Variable 


Moderating  Variable 


Intervening  Variable 


HI  and  H2  Hypothesis  Relationship 


For  Hypothesis  2,  a  correlation  was  used  to 
determine  the  strength  of  the  relationship  between 
experience  and  training  and  its  potential  causal 
relationship  to  knowledge. 

A  correlation  assessment  was  also  performed  to 
determine  the  presence  of  any  strong  connections 
among  gaps  in  knowledge,  training  and  experience. 


RESEARCH  QUESTIONS  AND  HYPOTHESES 


Research  Question  1 

•  What  are  the  relationships  among  training  gap, 
experience  gap,  and  performance  attributes  that 
affect  incentive  contract  arrangements  among 
Defense  Acquisition  Workforce  members  in  the 
contracting  and  program  field? 

•  Hypothesis  1:  There  are  gaps  in  training  and 
experience  that  affect  performance  attributes 
associated  with  incentive  contract  arrangements 
among  Defense  Acquisition  Workforce  members 
in  the  contracting  and  program  field. 

-  Hypothesis  1A:  There  is  a  reliable 
relationship  among  experience  gaps  and 
performance  attributes  among  contract 
professionals  and  program  managers. 

-  Hypothesis  IB:  There  is  a  reliable 
relationship  among  training  gaps  and 
performance  attributes  among  contract 
professionals  and  program  managers. 


Research  Question  2 

•  What  are  the  relationships  between  training 
gaps,  experience  gaps,  and  knowledge  gaps 
that  affect  incentive  contract  arrangements 
among  Defense  Acquisition  Workforce 
members  in  the  contracting  and  program  field? 

•  Hypothesis  2:  There  are  differences  in  the 
relationships  between  training  gaps,  experience 
gaps,  and  knowledge  gaps  associated  with 
incentive  contract  arrangements  among 
Defense  Acquisition  Workforce  members  in  the 
contracting  and  program  field. 

-  Hypothesis  2A:  There  is  a  reliable 
relationship  among  experience  gaps  and 
knowledge  gaps  among  contract 
professionals  and  program  managers. 

-  Hypothesis  2B:  There  is  a  reliable 
relationship  among  training  gaps  and 
knowledge  gaps  among  contract 
professionals  and  program  managers. 
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METHODOLOGY  -EMPIRICAL  DATA 


This  research  was  designed  to  measure  gaps;  and 
identify  any  correlations  among  knowledge, 
training,  experience,  and  performance  that  might 
prevail  with  incentive  and  award  fee  contracts. 

Question:  Was  the  incentive  contracting  training 
sufficiently  practical  and  comprehensive  to  enable 
PM/CM  to  achieve  program  outcomes  in  eight 
specific  performance  areas? 


Findings 

Recommendations 


Qualitative  Component: 

•  Over  6,000  comments 

•  Augmented  several  findings 


Question:  Were  there  any  training  and 
experience  gaps  that  impacted 
knowledge  and  performance  outcomes? 


2  Controlled  Groups  DAWIA  Level 

•  Program  Management  (PM) 

- 1194  PM  Participants 

•  Contracts  Management  (CM) 

-  946  CM  Participants 

•  70%  Civilian 

•  24%  Military 

•  6%  Contractors 


Quantitative  findings  centered  on: 

•  Training/Experience/Knowledge 

•  Presence  of  gaps 

•  Training/Experience  influence  on 
performance 


Program  Management  group  consisted  of  Program 
Managers,  Deputy  Program  Managers,  and  function 
acquisition  leads. 

Contracts  Management  group  consisted  of  Procuring 
Contracting  Officers,  Administrative  Contracting  Officers, 
and  Contract  Specialists. 

30  Question  Online  Survey 

Non-attribution  7 


EflJ  Findings:  Quantitative  Data 

•  Moderate  (self-assessed)  gaps  exist  in 
training,  experience,  and  Performance  (HI A/HI B). 

•  PMs  and  CMs/KOs  perceive  a  strong  relationship 
between  both  the  observed  training  gap  and 
experience  gap  to  the  observed  knowledge  gap. 

•  The  experience  gap,  vice  the  training  gap,  is  perceived  to  be 
more  closely  related  to  performance  outcomes. 

•  CMs  see  training  to  be  more  closely  related  to  performance 
outcomes  than  do  PMs. 

•  Training  and  experience  are  highly  correlated  to  knowledge. 

•  Current  state  of  skill-sets  is  more  about  the  mechanics 

•  Desired  state  focused  upon  the  performance  achieved  through 
drafting  incentives  to  achieve  desired  outcomes. 


3AU  Findings:  Quantitative  Data  (Cont’d) 

\  •  H2A/H2B  -  Incentive  contracting  Training(Tr)/Experience  (Ex) 

-  There  is  a  strong  correlation  to  Knowledge  (Kn). 

y  CMs  and  PMs  self-assessed  their  “current”  and 

levels  of  performance  achieve  through  incentive  contracts. 

-  CMs  (Current)  -Tr/Ex  gaps  exist;  Kn  is  limited  to  basic  mechanics  of  incentive; 

-  CMs  (Should  Be)  need  to  increase  Tr/Ex  to  structure  incentive  contracts  to  support 
more  in-depth  Kn  levels. 

-  PM  (Current)  assessments  of  CM  performance  level  is  closely  aligned  with  CM  self- 
assessments  and  corroborated  what  the  research  confirmed. 

-  PM  (Should-Be)  self-assessments  were  very  similar  to  CM  (Should  Be)  self- 
assessments. 

-  CM  (Should  Be)  assessments  of  PMs 

•  Except  -  Assessment  of  DPMs  -  lack  a  clear  understanding  of  incentive  knowledge. 

•  DPMs  has  a  2  point  gap  with  incentive  contract  formation  to  performance  outcomes. 

•  A  strong  relationship  exists  between  the  observed  training  and  experience 
gaps  and  knowledge  gaps — reducing  Tr/Ex  gaps  could  also  reduce  Kn  gaps. 


£AU  Gap  Analysis  (H1A/H1B) 


• 

.  L 

Table  8 

Results  of  a  Mean  Gap  Analysis  Between  Contract  Managers  Current  and  Should  Be 
Performance  Level 

Variables 

N 

Current 

Mean 

Current 

SD 

Should  be 

Mean 

Should 

be  SD 

Mean 

Gap 

Program  Managjer 

304 

3.0968 

1.7832 

4.1184 

1.7723 

1.0216 

Deputy  Program 
Manager 

301 

3.0370 

1.7899 

4.0864 

1.7318 

1.0494  | 

Contracting  Officer 

327 

4.0185 

2.0340 

5.1804 

1.5412 

1.1619 

Contract  Specialist 

369 

3.3425 

1.9019 

4.7015 

1.7621 

1.3590 

Table  8  -  Mean  gap  analysis  of  current  &  should  be 

performance  levels  by  CM  respondents. 

•  The  scale  measures  the  various  levels  of 
understanding  of  outcomes  that  can  be  achieved 
through  incentive  arrangements 

•  CM  respondents  identified  a  gap  greater  than  1 .0 
for  all  four  groups  (PM,  DPM,  KO,  &  CS). 

•  Greatest  gaps  were  CS  with  a  1 .3590  and  KO  with 
a  1.1619. 


Table  9 

Results  of  a  Mean  Gap  Analysis  Between  Program  Managers  Current 
and  Should  Be  Performance  Level 


Variables 

N 

Current 

Mean 

Current 

SD 

Should 

be  Mean 

Should 

be  SD 

Mean 

Gap 

Program 

Manager 

386 

3.0200 

1.7777 

4.1036 

1.7756 

1.0836 

Deputy 

Program 

Manager 

Contracting 

Officer 

343 

3.9510 

2.0372 

5.1590 

1.5382 

1.2080 

Contract 

327 

3.2416 

1.9019 

4.6812 

1.7604 

1.4396 

Specialist 


Table  9  -  Mean  gap  analysis  of  current  &  should  be 

performance  levels  by  PM  respondents. 

•  PM  respondents  identified  a  gap  greater  than  1 .0  for 
each  of  the  groups. 

•  Greatest  gaps  were  deputy  program  managers  with  a 
2.0384  and  contract  specialists  with  a  1 .4396. 

•  Program  managers  believe  that  among  DPMs  and 
CSs,  performance  levels  should  result  in  more 
meaningful  performance  outcomes. 
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SAU  Findings:  Qualitative  Data 

Knowledge: 

“Contractor  outcomes-  profit/fee-  were  higher  than  they  should 
have  been  because  personnel  routinely  failed  to  hold  contractors 
to  th»  r  '  Award  or  Incentive  plans.” 

they  don't  fully 


Over  6,000  comments  provided 
from  respondents. 


was  unable  to  write  the  uu 


to  use  FFP  or  CPFF  or  IDIQ  arrangements 
formation  of  award  fee  contracts.” 


•  “Not  sure  all  of  my  people  are  good  at  thinking  through  how  different  incentive 
structures  will  cause  the  contractor  to  behave.  Incentives  other  than  cost  are 
particularly  tricky  and  \  generally  shy  away  from  them  because  I  sense  that  the 
contractor  will  'outfox'  us  and  we  will  end  up  regretting  the  structure  down  the  road.” 


Findings:  Qualitative  Data 


Experience: 

•  “Experience  is  the  main  driver  of  shortfalls.  Not 
everybody  is  in  situations  where  they  are  using  CPAF, 

FPIF,  or  CPIF  contracts  regularly.” 

•  “This  lack  of  experience,  understanding  and  training 
makes  it  very  difficult  to  effectively  utilize  these  contract  types.” 

•  “Nobody  seems  to  know  what  to  do,  if  and  when  we  use  it  and/or  the  required 
information  is  not  passed  to  the  field. 

•  “Experience  seems  to  drive  individuals  to  contracts  that  they  are  familiar  with.” 

•  “Lack  of  experience  with  multiple  contract  types  can  cause  under-performance.” 

•  “The  lack  of  experience  outs  (place)  the  government  at  a  disadvantage  in  execution 
of  these  types  of  contracts.” 


«  54 


v^VNxjuu  uif 

for  Llie  i&i-E' 
experience  z 
,  new 

Mfj 


•  “There's  just  no  substitute  for  experience,  not  IQ,  not  education.  With  experience 
comes  intuition,  and  it's  intuition  that  recognizes  what  flies  and  what  doesn't. 
Oftentimes,  people  are  thrust  into  programs  and  projects  for  which  they  lack  a  basis 
for  making  informed  decisions  about  the  future.” 


3AU  Findings:  Qualitative  Data 

\  Training: 

•  “People  had  the  training,  but  did  not  understand 

how  to  use  it  in  their  duties.”  .  _  ,  _ _ 

•  Most  time  is  spent  trying  to  apply  mechanics  of  method  -  : 

type  rather  than  truly  implementing  meaningful  measures.” 

•  “The  biggest  problem  is  developing  meaningful  criteria. 

Services  acquisition  are  very  hard  to  'incentivize'.  The  wording  of  the 
meaningful  criteria  is  the  bigger  problem  than  coming  up  with  10,  20,  30 
percent  incentive.  That  is  simple  math.” 

•  “Because  we  don't  have  good  command  level  training  for  how  to  administer 
these  overly  complex  contracts,  everyone  struggles  to  efficiently  do  their 
work.” 

•  “Lack  of  training  is  the  biggest  problem.” 

•  “We  do  not  receive  enough  training  to  understand  these  concepts  enough  to 
execute  effectively.” 


Migj 


develop 

achieve 

help 


£AU  Recommendations 

•  Create  user-friendly  knowledge  sharing  access  for  lessons 
learned,  proven  practices,  successes,  etc.  DAU  Workflow. 

-  Use  as  resource  for  teaching  &  searchable  practicing 
opportunities  for  PMs/CMs 

•  Reinvigorate  incentive  contract  development  training  for  PMs  and  CMs 
so  they  better  understand  the  fundamental  principles  and  benefits  of 
incentive  contracts.  DAU  Develop  CLM  -  Incentive  Contracts. 

•  Establish  multiple  training  opportunities  for  incentive  contracts.  DAU 

-  Leverage  just-in-time  training  and  consulting  services 

-  Develop  assignment-specific  workshops  that  contain  rigorous 
exercises  and  fundamental  understanding  of  the  principles  and 
benefits  of  incentive  contracts 

-  Design  team  training  vignettes  using  available  activities  data 

•  Produce  an  incentive  and  award  fee  guidebook.  DAU/DPAP. 
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£AU  Recommendations  (Cont’d) 

•  Identify  the  lessons  learned  from  successful  and 
unsuccessful  incentive  arrangements  within  the  past 
10  years.  Workflow  Product. 

•  Ensure  the  appropriate  acquisition  workforce  qualification 
competencies  (or  any  variation  thereof)  incorporate  the  key 
standards  that  address  incentive  and  award  fee  contract  proficiencies 
that  are  sometimes  assignment  specific. 

•  Encourage  the  Functional  Integrated  Product  Teams  (FIPTs)  to  explore 
a  minimum  set  of  competencies  that  are  specifically  tuned  to  the 

implementation  of  incentive  arrangements  for  PMs  and  CMs  in 
assignment  specific  positions. 

•  Conduct  a  follow  on  study  to  address  the  relationship  between 
knowledge,  performance,  and  the  applicable  regulations. 
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Summary 


There  is  a  need  for  more  in  depth  incentive  contract  training  for 
PMs/KOs  to  support  assignments  specific  duties  in  support  of 
complex  programs  with  incentive  arrangements. 


•  CMs/PMs  need  additional  training,  hands  on  experience,  and  skill 
development  in  the  development  of  complex  incentive  and  award 
fee  arrangements  that  specifically  target  performance  outcomes. 


Any  incentive  arrangement  training  needs  to  include  information 
on  both  the  development  and  application  of 
objective  incentives  to  achieve  program 
outcomes  while  rewarding  the  contractor 
when  program  goals  are  achieved. 


BBP  encourages  the  use  of  incentive  arrangements. 


